Time-Varying Gaussian Process Bandit Optimization

We consider the sequential Bayesian op-timization problem with bandit feedback,adopting a formulation that allows for the re-ward function to vary with time. We modelthe reward function using a Gaussian pro-cess whose evolution obeys a simple Markovmodel. We introduce two natural extensionsof the classical Gaussian process upper confi-dence bound (GP-UCB) algorithm. The first,R-GP-UCB, resets GP-UCB at regular in-tervals. The second, TV-GP-UCB, insteadforgets about old data in a smooth fashion.Our main contribution comprises of novel re-gret bounds for these algorithms, providingan explicit characterization of the trade-offbetween the time horizon and the rate atwhich the function varies. We illustrate theperformance of the algorithms on both syn-thetic and real data, and we find the gradualforgetting of TV-GP-UCB to perform favor-ably compared to the sharp resetting of R-GP-UCB. Moreover, both algorithms signifi-cantly outperform classical GP-UCB, since ittreats stale and fresh data equally.